Data-agnostic adjustment of hard thresholds based on user feedback

ABSTRACT

This disclosure is directed to data-agnostic computational methods and systems for adjusting hard thresholds based on user feedback. Hard thresholds are used to monitor time-series data generated by a data-generating entity. The time-series data may be metric data that represents usage of the data-generating entity over time. The data is compared with a hard threshold associated with usage of the resource or process and when the data violates the threshold, an alert is typically generated and presented to a user. Methods and systems collect user feedback after a number of alerts to determine the quality and significance of the alerts. Based on the user feedback, methods and systems automatically adjust the hard thresholds to better represent how the user perceives the alerts.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 14/312,815,filed Jun. 24, 2014.

TECHNICAL FIELD

The present disclosure is directed to adjusting hard thresholds oftime-series data based on user feedback.

BACKGROUND

In recent years, the number of enterprises relying on cloud computing tomeet their computing needs has substantially increased. Many enterprisesare able to cut costs by simply purchasing cloud computing services fromhosting service providers that maintain cloud computing facilities. As aresult, these enterprises eliminate a heavy investment in facilities,security, upgrades, and operating expenses. Cloud computing is typicallycarried out in computing facilities that house a vast array of networkedphysical machines (“PMs”), data-storage devices, and network routers.The facilities use virtualization to efficiently and cost effectivelyrun computing processes on one or more connected PMs. Withvirtualization, one or more PMs are partitioned into multipleindependent virtual machines (“VMs”) that function independently andappear to users as actual PMs. VMs can be moved around and scaled up ordown as needed without affecting the user's experience.

In order to maintain computing facility operations and execution ofapplications, many physical and virtual computational resources, such asprocessors, memory, and network connections, and other data-generatingentities are monitored over time. Data-generating entities generatetime-series data that is collected, analyzed, and presented for humanunderstanding. An alert is typically generated when the data violates ahard threshold so that a user can identify anomalies. However, becausehard thresholds are static while data-generating entities may changeover time, the likelihood of generating a false positive alert (i.e., analert that incorrectly indicates a problem) or a false negative alert(i.e., an alert not given when there is a problem) based on the hardthresholds increases.

SUMMARY

This disclosure is directed to data-agnostic computational methods andsystems for adjusting hard thresholds based on user feedback. Hardthresholds are used to monitor time-series data generated by adata-generating entity. The data-generating entity can be acomputational process, computer, sensor, virtual or physical machinerunning in a data center or other computational environment, or acomputational resource, such as a processor, memory, or networkconnection. The time-series data may be metric data that representsusage of the data-generating entity over time. The time-series data iscompared with a hard threshold and when the data violates the threshold,an alert is typically generated and presented to a user. Methods andsystems collect user feedback after a number of alerts to determine thequality and significance of the alerts. Based on the user feedback,methods and systems automatically adjust the hard thresholds to betterrepresent how the user perceives the alerts.

DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a general architectural diagram for various types ofcomputers.

FIG. 2 shows a plot of time-series data generated for a physical orvirtual computational resource.

FIG. 3 shows an example of survey questions that may be presented to auser after an alert.

FIG. 4 shows an example plot of feedback statistics collected after sixalerts.

FIG. 5 shows a plot of two exponential weights functions.

FIG. 6 shows an example distribution/histogram of weighted statisticsbinned into five subintervals.

FIG. 7 shows an example table of differently rated alerts.

FIG. 8 shows a plot of time-series data with a lower hard threshold.

FIG. 9 shows a weight statistic histogram with lower bound of the modalinterval identified.

FIG. 10 shows a weight statistic histogram with a lower bound of aninterval identified.

FIG. 11 shows a plot of time-series data with a higher hard threshold.

FIG. 12 shows a flow-control diagram of a method for adjusting a hardthreshold.

FIG. 13 shows a flow-control diagram of the routine “calculateconfidences” called in block 1207 of FIG. 12.

FIG. 14 shows a flow-control diagram of the routine “adjust alertthresholds” called in block 1211 of FIG. 12.

FIG. 15 shows a flow-control diagram for the routine “move hardthreshold down” called in block 1404 of FIG. 14.

FIG. 16 shows a flow-control diagram for the routine “move hardthreshold up” called in block 1406 of FIG. 14.

FIG. 17 shows a flow-control diagram for the routine “adjustcriticality” called in block 1408 of FIG. 14.

FIG. 18 shows a flow-control diagram of the routine “adjust alertthresholds” called in block 1211 of FIG. 12.

FIG. 19 shows a flow-control diagram for the routine “move hardthreshold up” called in block 1801 of FIG. 18.

FIG. 20 shows a flow-control diagram for the routine “move hardthreshold down” called in block 1802 of FIG. 18.

DETAILED DESCRIPTION

This disclosure presents data-agnostic computational systems and methodsfor adjusting hard thresholds used to monitor time-series data based onuser feedback. It should be noted, at the onset, that the currentlydisclosed computational methods and systems are directed to real,tangible, physical systems and the methods carried out within physicalsystems, including client computers and server computers. Those familiarwith modern science and technology well appreciate that, in moderncomputer systems and other processor-controlled devices and systems, thecontrol components are often fully or partially implemented as sequencesof computer instructions that are stored in one or more electronicmemories and, in many cases, also in one or more mass-storage devices,and which are executed by one or more processors. As a result of theirexecution, a processor-controlled device or system carries out variousoperations, generally at many different levels within the device orsystem, according to control logic implemented in the stored andexecuted computer instructions. Computer-instruction-implemented controlcomponents of modern processor-controlled devices and systems are astangible and physical as any other component of the system, includingpower supplies, cooling fans, electronic memories and processors, andother such physical components.

FIG. 1 provides a general architectural diagram for various types ofcomputers. The internal components of many small, mid-sized, and largecomputer systems as well as specialized processor-based storage systemscan be described with respect to this generalized architecture, althougheach particular system may feature many additional components,subsystems, and similar, parallel systems with architectures similar tothis generalized architecture. The computer system contains one ormultiple central processing units (“CPUs”) 102-105, one or moreelectronic memories 108 interconnected with the CPUs by aCPU/memory-subsystem bus 110 or multiple busses, a first bridge 112 thatinterconnects the CPU/memory-subsystem bus 110 with additional busses114 and 116, or other types of high-speed interconnection media,including multiple, high-speed serial interconnects. The busses orserial interconnections, in turn, connect the CPUs and memory withspecialized processors, such as a graphics processor 118, and with oneor more additional bridges 120, which are interconnected with high-speedserial links or with multiple controllers 122-127, such as controller127, that provide access to various different types of computer-readablemedia, such as computer-readable medium 128, electronic displays, inputdevices, and other such components, subcomponents, and computationalresources. The electronic displays, including visual display screen,audio speakers, and other output interfaces, and the input devices,including mice, keyboards, touch screens, and other such inputinterfaces, together constitute input and output interfaces that allowthe computer system to interact with human users. Computer-readablemedium 128 is a data-storage device, including electronic memory,optical or magnetic disk drive, USB drive, flash memory and other suchdata-storage devices. The computer-readable medium 128 can be used tostore machine-readable instructions that encode the computationalmethods described below and can be used to store encoded data, duringstore operations, and from which encoded data can be retrieved, duringread operations, by computer systems, data-storage systems, andperipheral devices.

Thresholds and User Input

FIG. 2 shows a plot of time-series data generated for a data-generatingentity. The data-generating entity can be a computational process,computer, sensor, virtual or physical machine running in a data centeror other computational environment, or a computational resource, such asa processor, memory, or network connection. The time-series data may bemetric data that represents usage of the data-generating entity overtime. Horizontal axis 202 represents time and vertical axis 204represents data values. Dots, such as dot 206, are data values thatrepresent usage of the resource measured at regularly intervals of time,and a curve 208 connecting the data values illustrates how the datavalues, or usage of the resource, changes over time. Horizontal line 210represents maximum usage of the resource. For example, the time-seriesof data 208 can represent processor usage by a VM, memory usage by a VM,amount of electrical power consumed by a VM, or hard-disk space used bya VM and line 210 may represent 100% usage of the resource. The data canalso represents usage of various physical resources of a data center,including buffer access, amount of memory in use, network connectionsused or idle, electrical power consumption, network throughput,availability of hard-disk space, and processor time.

In the example of FIG. 2, a user selects an upper hard threshold value,D, represented by dashed line 212 and a lower hard threshold, d,represented by dashed line 214. In particular, a user may only select anupper hard threshold, or a lower hard threshold, depending on theresource and the type of data. As shown in FIG. 2, the time-series ofdata 208 has three sets of consecutive data 216-218 with values greaterthan the hard threshold D and two sets of consecutive data 220 and 221with values less than the hard threshold d. The user also selects a waitinterval w that is used to generate an alert and a cancellation intervalc that is used to timely cancel the alert. The length of the waitinterval w is an integer that represents the minimum number ofconsecutive data points that violate the hard threshold beforegenerating an alert. The length or duration of the cancellation intervalc is an integer that represents the minimum number of consecutive datapoints returned to normal (i.e., no longer violate the threshold) afterthe last abnormality reported as an alert. The duration of thecancellation interval c determines when an active alert is canceled duebased on recovery. For example, FIG. 2 shows example wait intervals thatare three consecutive data points long, such as interval 222, andexample cancellation intervals that are five consecutive data pointslong, such as interval 224. When a number of consecutive data pointsthat violate a hard threshold is greater than or equal to the length w,an alert is generated, but the alert is cancelled and not reported tothe user when the number of consecutive data points returned to normalis greater than the length c. On the other hand, when the number ofconsecutive data points returned to normal is less than the length c,the alert is not cancelled and presented to the user. For example, inFIG. 2, the set 216 has only two consecutive data points that violatethe threshold D, which is not enough data points to generate an alert.The set 217 has four consecutive data points that violate the thresholdD, which is enough data points to generate an alert, but the alert iscancelled because the number of consecutive data points returned tonormal after the data point 226 is greater than the length c. On theother hand, the set 218 has eight consecutive data points that violatethe threshold D, which is enough data points to generate an alert. Butthis alert is not cancelled because only three consecutive data pointsafter the point 228 are below the threshold D which is less than thelength c. In the case of the two sets 220 and 221, alerts are alsogenerated because the number of consecutive data points in each set isgreater than the length w, but the alerts are cancelled because thenumber of consecutive data points following the sets 220 and 221 isgreater than the length c.

The user also selects an alert criticality level L to assign a level ofimportance to an alert. The criticality level L is a number in theinterval [0,1]. When L=0 the alert is “non-critical” or “none,” when thecritical L=0.25 the alert is “informative;” when the criticality isL=0.5 the alert is a “warning;” when the criticality is L=0.75 the alertis “immediate;” and when criticality is L=1 the alert is “critical.”

After an alert is displayed for a user, the user is presented withsurvey questions to determine how indicative the alert was of a problemwith usage of the resource. Answers to the survey questions form userfeedback that is used as input to adjust the hard threshold. However,adjustments to the hard threshold, as described below, are controlled bya user-defined noise tolerance N. The noise tolerance N is a numericalvalue in the interval [0,1] selected by a user to represent the user'stolerance to false positive alerts. A noise tolerance N equal to “0”indicates the user has no tolerance for false positive alerts while anoise tolerance N equal to “1” indicates the user is indifferent tofalse positive alerts. For example, a user may select the noisetolerance N equal to 0.2, which indicates the user has a low toleranceto false positive alerts.

Collecting Feedback Statistics

FIG. 3 shows an example of survey questions that may be presented to auser after an alert has been generated. In the example of FIG. 3, theuser is presented with a general survey question regarding “Howindicative of a problem was the alert?” 302. The user may then selectone of five answers 304 that indicate the user's level of satisfactionwith the alert. The survey also includes three additional more specificquestions regarding “How indicative was the alert in terms of?” 306“Criticality” 308, “Timeliness” 310, and “Duration” 312. For each of theexample questions, the user selects one of five answers that indicatethe user's level of satisfaction with the indicativeness, criticality,timeliness, and duration of the alert. The five answers the user mayselect from to answer each question are associated with numerical valuesin parentheses that lie in the interval [0,1]. These numerical valuesform the feedback statistics. For example, with regard to question 1,the user has filled in the bubble 314 which indicates that the userfound the alert “rather” indicative of a problem which, in turn,corresponds to a feedback statistic of 0.5 316. Methods for adjustinghard thresholds are predicated on the assumption that the indicativenessof alert increases with the greater the distance a data value is from ahard threshold. For example, consider two sets of consecutive datavalues that violate the same hard threshold. It is assumed that the userwill rate the indicativeness of the alert associated with the setlocated farther from the hard threshold more critical than theindicativeness of the alert for the set located closer to the hardthreshold.

For this particular example survey questions in FIG. 3, the feedbackstatistic have a feedback resolution of five, which corresponds to thefive ways the user may answer each question. A survey questions withonly two possible answers, such as “like” (1.0) of “dislike” (0.0)answers, represents the minimum in user feedback because there are onlytwo ways a user may indicate their level of satisfaction. In this case,the feedback resolution is two with no intermediate values that may usedto indicate varying degrees of user satisfaction.

Alternatively, indirect collection of feedback statistics may beobtained by tracking a user's activities for each alert. Any indirectfeedback that can be tracked over time may also be mapped to values inthe interval [0,1]. For example, a user's activities after an alert manybe monitored and certain actions counted and normalized to determinefeedback statistics for each alert.

Methods for Calculating Confidence and Weighted Average of FeedbackStatistics

In a data-agnostic approach to adjusting a hard threshold, beliefs areapplied directly without user experience or expertise of direct orindirect feedback consideration. Consider a set of beliefs associatedwith a user's assessment of an alert given by:

B={B _(al) ,B _(crit) ,B _(time) ,B _(dur)}  (1)

where

-   -   B_(al) represents a belief about the indicativeness of the        alert;    -   B_(crit) represents a belief about the criticality of the alert;    -   B_(time) represents a belief about timeliness of the alert        (i.e., wait interval); and    -   B_(dur) represents a belief about the duration of the alert        (i.e., cancellation interval).        In the follow description, each belief in the set B is        represented by B_(i), where the index i represents “al,” “crit,”        “time,” and “dur.” Each belief B_(i) represents a statement,        truth, law, or expert knowledge about an alert presented to a        user or any statement, truth, law, or expert knowledge learned        data agnostically about an alert presented to a user. The        beliefs may also be represented by probabilities. For example,        each belief in the set B may be represented by a value in the        interval 0≦B_(i)≦1, with “1” representing a maximum confidence        in a statement, truth, law, or expert knowledge about an alert        presented to a user, and “0” representing no confidence in a        statement, truth, law, or expert knowledge about the alert        presented to the user.

Feedback statistics for the belief B_(i) are collected after each alertto form a set of feedback statistics

F(B _(i))≡{f ₁(B _(i)), . . . ,f _(K)(B _(i))}={f _(k)(B _(i))}_(k=1)^(K)  (2)

where

-   -   subscript k is an integer feedback statistic index;    -   f_(k)(B_(i)) is the k-th feedback statistic for the belief        B_(i); and    -   K is an integer number of feedback statistics.        Each feedback statistic f_(k)(B_(i)) in the set of feedback        statistics corresponds to a value in the interval [0,1]. For        example, the k-th feedback statistics for the answers to the        survey questions in FIG. 3 are f_(k)(B_(al))=0.5,        f_(k)(B_(crit))=0.5, f_(k)(B_(time))=1.0, and        f_(k)(B_(dur))=0.25. In other words, four sets of feedbacks        statistics F(B_(al)), F(B_(crit)), F(B_(time)), and F(B_(dur))        are generated for K alerts. Because the feedback statistics are        collected at different times, the feedback statistics may also        be considered a collection of time-dependent feedback statistics        denoted by

F(B _(i))≡{f(t _(k) ,B _(i))}_(k=1) ^(K) ={f _(k)(B _(i))}_(k=1)^(K)  (3)

where t_(k) represents the time at which the feedback statistics wheregenerated.

FIG. 4 shows an example plot of feedback statistics collected after sixdifferent alerts. Vertical axis 402 represents feedback statisticsvalues in the interval [0,1]. Axis 404 represents k, and axis 406represents the beliefs B_(al), B_(crit), B_(time), and B_(dur). Barsextending perpendicular from the k-beliefs plane represent feedbackstatistics associated with each belief. For example, bar 408 representsthe feedback statistic f₆(B_(dur)). The varying height of the bars asindicated by dashed lines, such as dashed lines 410, represent howfeedback statistics for a particular belief may vary after each alert.For example, the feedback statistic associated with the belief B_(al) istrending down, which indicates that a user finds the alerts lessindicative of a problem over time.

Based on the set of feedback statistics F(B_(i)) a convergenceevaluation in user opinion is made and a confidence value C(B_(i)) iscalculated. The confidence value C(B_(i)) supports the degree ofvalidity of the initial belief B_(i). The method used to adjust a hardthreshold, criticality, wait interval, and cancellation intervaldescribed below is predicated on three postulates:

1) The posting of feedback statistics is assumed to be a process withincreasing degree of importance with respect to time (in particular, anindependent and identically distributed process);

2) When there is no convergence in user feedback statistics, the hardthreshold, criticality, wait interval, and cancellation interval are notupdated;

3) When there is a convergence to some degree of user feedback, the hardthreshold, criticality, wait interval, and cancellation interval areadjusted according to the corresponding calculated confidence values.

The feedback convergence is estimated by processing the feedbackstatistics with weighted importance based on time and measuring theuncertainty. In other words, if the confidence is low enough, a bias inweighted opinion statistics is estimated. Weighted statistics of a pastseries of feedback statistics may be calculated at each time t_(r) whenan alert is generated as follows:

$\begin{matrix}{{S\left( {f_{k}\left( B_{i} \right)} \right)} = \frac{\sum_{r = 1}^{k}{{w\left( t_{r} \right)}{f_{r}\left( B_{i} \right)}}}{\sum_{r = 1}^{k}{w\left( t_{r} \right)}}} & (4)\end{matrix}$

where w(t_(r)) is a weight function.

The weighted statistic values lie within the interval [0,1] (i.e.,0≦S(f_(k)(B_(i)))≦1). The weight function ranges from 0 to 1 over a timedomain 0 to t_(k). The weight function is selected to place more weightor influence on feedback statistics collected later in time than onfeedback statistics collected earlier in time. In other words, theweighted statistic given by Equation (4) is a time-dependent weightedmean of the feedback statistics collected over time between 0 and t_(k)with more weight placed on feedback statistics collected later in time.For example, the weight function is selected to give the feedbackstatistic f_(y)(B_(i)) more weight in Equation (4) than the feedbackstatistic f_(x)(B_(i)), where 0≦t_(x)<t_(y)≦t_(k). An example of aweight function w(t_(r)) that places more weight on feedback statisticscollected later in time is an exponential weight function given by:

$\begin{matrix}{{w\left( t_{r} \right)} = \left\{ \begin{matrix}1 & {{{for}\mspace{14mu} r} = k} \\e^{- {({t_{k} - t_{r}})}} & {{{for}\mspace{14mu} r} < k}\end{matrix} \right.} & (5)\end{matrix}$

Alternatively, another example of a weight function w(t_(r)) that placesmore weight on feedback statistics collected later in time is a linearweight function given by:

$\begin{matrix}{{w\left( t_{r} \right)} = {\frac{1}{t_{k}} \cdot t_{r}}} & (6)\end{matrix}$

where 0≦t_(r)≦t_(k). In an alternative implementation, the feedbackstatistics may all be given equal weight (i.e., w(t_(r))=1).

FIG. 5 shows a plot of the exponential weight function in Equation (5)and a plot of the linear weight function in Equation (6). Horizontalaxis 502 represents time t_(r) and vertical axis 504 represents thevalue of the weight function w(t_(r)), which ranges from 0 to 1. Curve506 represents the exponential function given by Equation (5), anddashed line 508 represents the linear function given by Equation (6).Both weight functions are 0 at time 0 and increase to a value of 1 attime t_(k). In other words, the weight functions represented byEquations (5) and (6) place more weight on feedback statistics collectedlater in time than on feedback statistics collected earlier in time withthe most current weight w(t_(k)) having a value of 1.

A set of weighted statistics obtained over a time interval from 0 tot_(K) is given by

S (B _(i))≡{S(f ₁(B _(i))), . . . ,S(f _(K)(B _(i)))}  (7)

The weighted statistic values range over the interval [0,1], which isdivided into l subintervals. The weighted statistics in the set ofweighted statistics S(B_(i)) are binned according to which subintervalof the interval [0,1] the weighted statistics values fall within. Thenumber of subintervals l of the interval [0,1] corresponds to theresolution of the requested feedback statistics. For example, if binarylike/dislike user feedback is expected, then l=2 is selected. In thiscase, the feedback statistics may be 0 or 1 and the interval [0,1] maybe partitioned into two subintervals [0,0.5) and [0.5,1]. On the otherhand, if 5 possible feedback statistics are expected, as described abovewith reference to FIG. 3, then l=5 is selected. In this case, thefeedback statistics may be 0, 0.25, 0.5, 0.75, and 1, as described abovewith reference to the example survey question of FIG. 3, and theweighted statistics fall into five subintervals [0,0.20), [0.20,0.40),[0.40,0.60), [0.60,0.80), and [0.80,1]. Note that the subintervals donot have to be of the same length.

FIG. 6 shows an example distribution/histogram for 40 weightedstatistics for the belief B_(al) binned into five subintervals (i.e.,l=5) of the interval [0,1]. Horizontal axis 602 represents the weightedstatistical values in the interval [0,1], and vertical axis 604represents the frequency or count of the weighted statistics within thefive subintervals identified by subinterval index r=1, 2, 3, 4, and 5.Boxes, such as box 606, represent 40 weighted statistics in setS(B_(al)) (i.e., K=40) binned according to which subinterval theweighted statistic falls within. For example, six of the 40 weightedstatistic values in the set S(B_(al)) lie within subinterval r=1.

The fraction of weighted statistics that lie within each subinterval ofthe histogram of weighted statistic in FIG. 6 are normalized frequenciesdenoted by h_(r). In other words, in general, Σ_(r=1) ^(l) h_(r)=1. Forexample, in FIG. 6, the normalized frequencies for each of thesubintervals are given by:

$\left\{ {h_{1},h_{2},h_{3},h_{4},h_{5}} \right\} = \left\{ {\frac{6}{40},\frac{9}{40},\frac{7}{40},\frac{11}{40},\frac{7}{40}} \right\}$

Uncertainty in the weighted statistics of Equation (4) may be determinedby calculating the entropy of the normalized frequencies:

$\begin{matrix}{{H\left( {\overset{\_}{S}\left( B_{i} \right)} \right)} = {- {\sum\limits_{r = 1}^{l}{h_{r}\log_{l}h_{r}}}}} & (8)\end{matrix}$

where

Σ_(r=1) ^(l) h _(r)=1.

Note that the entropy calculated according to Equation (8) satisfies thecondition

0≦H( S (B _(i)))≦1

Next, confidence in a belief B_(i) is calculated based on the entropy.When the entropy H(S(B_(i))) is less than or equal to an uncertaintythreshold denoted by U_(th) (i.e., H(S(B_(i)))≦U_(th)), the uncertaintyin the feedback statistics associated with the belief B_(i) is low andthe confidence in the belief B_(i) may be calculated as a function ofthe entropy as follows:

C(B _(i))=1−H( S (B _(i)))  (9)

On the other hand, when the entropy H(S(B_(i))) is greater than theuncertainty threshold U_(th) (i.e., H(S(B_(i)))>U_(th)), the uncertaintyin the feedback statistics associated with the belief B_(i) is high andthe confidence is given by:

C(B _(i))=0  (10)

An example of a suitable uncertainty threshold value is:

$\begin{matrix}{U_{th} = {{{- \frac{1}{3}}\log_{l}\frac{1}{3}} - {\frac{2}{3}\log_{l}\frac{2}{3}}}} & (11)\end{matrix}$

The uncertainty threshold characterized by Equation (11) corresponds toa histogram of weighted statistics in which l−2 subintervals of theinterval [0,1] contain 0 weighted statistics and two other subintervalshave ⅓ and ⅔ of the weighted statistics, respectively, which is a caseof acceptable uncertainty in feedback statistics. When the entropy isgreater than U_(th), there is no convergence in user opinion and thecorresponding confidence should be at the minimum (i.e., C(B_(i))=0)).Because the entropy is normalized, in alternative implementations theuncertainly threshold U_(th) may be assigned a value in the interval(½,1] (e.g., a value close to 1).

The average of the weighted statistics in the set S(B_(i)) is calculatedaccording to

$\begin{matrix}{{m_{i}\left( h_{\max} \right)} = {\frac{1}{K}{\sum\limits_{k = 1}^{K}{S\left( {f_{k}\left( B_{i} \right)} \right)}}}} & (12)\end{matrix}$

The mode of the histogram of weighted statistics is given by

h _(max)=max{h ₁ , . . . ,h _(i)}  (13)

In other words, the mode h_(max) of the histogram of the set S(B_(i)) isthe largest normalized frequency of weighted statistics and correspondsto the subinterval, called the “modal subinterval,” which is thesubinterval with the largest number of weighted statistics. For example,in FIG. 6, the mode h_(max) is h₄=11/40 and the modal subinterval isr=4. The mode h_(max) takes into account the degree of importance intime of the weighted statistic values that lie within the modalsubinterval of the histogram. When the uncertainty H(S(B_(i))) is lessthan or equal to the uncertainty threshold U_(th), the confidenceC(B_(i)) is calculated according to Equation (9) by checking whichsubinterval contains the bias in uncertainty. The subinterval with thelargest bias in uncertainty corresponds to the mode of the histogram,h_(max).

Methods for Calculating an Adjusted Hard Threshold

Feedback-based adjustments of a hard threshold may be executed when asufficient number of alerts with feedback statistics have been collectedfrom a user and a sufficient number of feedback statistics associatedwith other users have been collected. In particular, methods foradjusting a hard threshold may be executed when the following twoconditions are satisfied:

1. The minimum number of alerts with feedback statistics is a; and

2. At least 25% of users produced feedback statistics that satisfy

$\begin{matrix}{{b\frac{f}{U}} \leq {f_{u}}} & (14)\end{matrix}$

where

-   -   U is the total number of users;    -   ∥f∥ is the number of all available feedbacks;    -   ∥f_(u)∥ is the number of feedbacks generated by user u; and    -   b>0 is parameter with default value 1 that may be set to require        at least b-times the average feedback per user (i.e., ∥f∥/U) for        some portion of the users.

Assuming the two conditions for having enough feedback statisticsassociated with other users are satisfied, the method then proceeds todetermining whether or not a hard threshold should be adjusted. Consideradjusting an upper hard threshold D, such as the upper hard threshold D212 described above with reference to FIG. 2. The confidence C(B_(al))is calculated from the feedback statistics F(B_(al)) according toEquation (9). When the confidence C(B_(al)) equals zero, the upper hardthreshold is not adjusted. On the other hand, when the confidenceC(B_(al)) is greater than zero, the average m_(al)(h_(max)) of the setof weighted statistics S(B_(al)) is calculated according to Equation(12).

Next, the user noise tolerance N is compared with a noise degree at theuser (i.e., a noise degree) quantified by

(1−m _(al)(h _(max)))  (15)

The noise degree is an estimate of the actual noise degree that thefeedback statistics obtained from the users indicate. The noise degreeis equal to “0” when all alerts are rated perfectly. Otherwise, thenoise degree indicates a departure from perfection. The noise degree iscompared with the user's selected noise tolerance N to determine whetheror not the alerts generated by violating the hard threshold D satisfythe user's tolerance for false positive alerts. In particular, when thedifference between the noise degree and the user's noise tolerance Nsatisfies the following condition

|(1−m _(al)(h _(max)))−N|≦δ  (16)

with a tolerance parameter δ (e.g., δ=0.01), the noise degree(1−m_(al)(h_(max))) is sufficiently close to the user noise tolerance Nthat the hard threshold D is left unchanged. Alternatively, when thedifference satisfies the following condition

(1−m _(al)(h _(max)))−N<−δ  (17)

the noise degree is too low, or when the difference satisfies thefollowing condition

(1−m _(al)(h _(max)))−N>δ  (17)

the noise degree is too high. When one of the conditions represented byEquations (17) and (18) is satisfied, the hard threshold D is adjusted.

Consider the case where the noise degree (1−m_(al)(h_(max))) satisfiesthe condition represented by Equation (17). As a result, the hardthreshold is decreased to a lower hard threshold, which may trigger oneor more alerts from the time-series data that were not generated for theprevious hard threshold. Because feedback statistics were not generatedfor alerts triggered as a result of decreasing the hard threshold to alower hard threshold for the existing time-series of data, estimatedfeedback statistics regarding indicativeness of newly created/appearedalerts for the existing time-series of data are extrapolated fromfeedback statistics generated by the users for rated alerts based on theprevious threshold value. As a result, the feedback statistics generatedby users for the rated alerts are collected into an array.

FIG. 7 shows an example of a table 700 of differently rated alerts.Column 702 is a list alert indices and column 704 is a list of thenumber of feedbacks or number of ratings collected for each alert. Forexample, the number of feedbacks collected for the m-th alert 706 isrepresented by num_rating(m) 708, and the number of feedbacks collectedfor the n-th alert 710 is represented by num_rating(n) 712.

After the feedback statistics have been assembled into an array, anaverage of feedbacks counts per alert is calculated as follows:

$\begin{matrix}{{n(f)} = \frac{\sum_{m = 1}^{M}{{num\_ ratings}(m)}}{M}} & (19)\end{matrix}$

where M is the number of alerts ever rated; and

-   -   num_ratings(m) is the number of feedback statistics generated        for the m-th alert.        The average of feedback counts n(f) is rounded to its integer        part (i.e., truncated). Alternatively, the average of feedback        counts may be rounded to its nearest integer.

Next, the hard threshold D is iteratively decreased by initially settinga lower hard threshold D_(low) equal to the hard threshold D and, foreach iteration, calculating the lower hard threshold according to

D _(low) =D _(low)−ε  (20)

where ε>0 is the step size that can be even in precision of nearestneighbor data point down.

For each iteration that the lower hard threshold is decreased, thenumber of alerts generated from the existing time-series of datapotentially increases.

FIG. 8 shows the plot of time-series data shown in FIG. 2 with the hardthreshold decreased to lower hard threshold represented by dashed line802. As a result, a new set of consecutive data 804 is greater than thelower hard threshold 802 with the number of consecutive data pointsgreater than the length w, which generates a new alert. The new alert ismaintained because the number of consecutive data points below the lowerhard threshold is less than the duration c. As a result, estimatedfeedback statistics regarding indicativeness of the new alert associatedwith the set 804 are extrapolated from feedback statistics associatedwith the real rated alerts.

When a new alert is generated as a result of a lower hard threshold,estimated feedback statistics for the alert belief B_(al) are formedfrom lower bounds, V_(al)(r_(min)), of the model subintervals theweighted statistic histograms. For example, FIG. 9 shows the weightstatistic histogram for belief B_(al) shown in FIG. 6. As describedabove with reference to FIG. 6, the subinterval r=4 is the modalsubinterval with the largest number of weighted statistics counts at 11.The lower bound of the subinterval r=4 is denoted by V(r_(min)). Thelower bound V(r_(min)) is added as a feedback statistics n(f) times tothe set of feedback statistics.

The estimated feedback statistics for the new alerts are added to theset of feedback statistics F(B_(al)) represented by Equation (2) to givean enlarged set of feedback statistics

F _(K) (B _(al))≡{f ₁(B _(al)), . . . ,f _(K)(B _(al)),f _(K+1)(B_(al)), . . . ,f _(K) (B _(al))}  (21)

where f_(K+1)(B_(al))=V(r_(min)) for 1≦l≦K; and

-   -   K=K+K_(new)n(f), where K_(new) is the number of newly appeared        hypothetical alerts.        Weighted statistics for the set F _(K) (B_(al)) are calculated        according to Equation (4) to give a set of weighted statistic        given by

S _(K) (B _(al))≡{S(f ₁(B _(al))), . . . ,S(f _(K) (B _(al)))}  (22)

The average of the weighted statistics in the set S _(K) (B_(al)) iscalculated according to

$\begin{matrix}{{m_{al}\left( h_{\max} \right)} = {\frac{1}{\overset{\_}{K}}\left( {\sum\limits_{l = 1}^{\overset{\_}{K}}{S\left( {f_{l}\left( B_{al} \right)} \right)}} \right)}} & (23)\end{matrix}$

A noise degree (1−m_(al)(h_(max))) is calculated for the belief B_(al)and compared with the user's noise tolerance N. When the condition givenby Equation (17) is satisfied, the hard threshold is lowered againaccording to Equation (20) and the operations described for Equations(21)-(23) are repeated for the lower hard threshold. On the other hand,when the condition given by Equation (16) is satisfied or when a noisedegree maximum close to N is reached a fixed number of iterations P, theadjusted hard threshold is calculated according to

D=(1−C(B _(al)))D+C(B _(al))D _(low)  (24)

Alternatively, if condition given by Equation (16) is still notsatisfied and if

m _(al)(h _(max))−V(r _(min))≦δ  (25)

then V(r_(min)) is reset to the lower bound of the lesser valuedsubinterval adjacent to the modal interval. For example, FIG. 10 showsthe weight statistic histogram for belief B_(al) shown in FIG. 6. Asdescribed above with reference to FIG. 6, the subinterval r=4 is themodal subinterval with the largest number of weighted statistics countsat 11. The lesser valued subinterval adjacent to the modal subintervalr=4 is the subinterval r=3. The lower bound of the lesser valuedsubinterval r=3 is identified by V(r_(min)).

For each iteration in which the hard threshold is lowered according toEquation (20), an alert confidence C(B_(al)) is calculated. A weightedstatistics histogram is formed from the set of weighted statistics S_(K) (B_(al)) which gives a set of normalized frequencies {h′_(r)}determined from the l subintervals of the histogram. The entropy iscalculated for the normalized frequencies according to

$\begin{matrix}{{H\left( {{\overset{\_}{S}}_{\overset{\_}{K}}\left( B_{al} \right)} \right)} = {- {\sum\limits_{r = 1}^{l}{h_{r}^{\prime}\log_{l}h_{r}^{\prime}}}}} & (26)\end{matrix}$

When H(S _(K) (B_(al)))≦U_(th), the confidence includes contributionsfrom feedback statistics associated with the new alerts is calculatedaccording to

C(B _(al))=1−H( S _(K) (B _(al)))  (27)

Otherwise, the confidence is given by

C(B _(al))=0  (28)

When the alert confidence C(B_(al)) is greater than zero (i.e.,C(B_(al))>0), the noise degree is calculated according to Equation (15)using the average of the weighted statistics given by Equation (27). Ifthe noise degree satisfies the condition given by Equation (17), theiteration stops. Otherwise, the iteration stops for the maximum noisedegree estimate closest to N.

Consider the case in which the noise degree (1−m_(al)(h_(max)))satisfies the condition represented by Equation (18). In this case, thehard threshold D is iteratively increased. Initially, a higher hardthreshold D_(high) is set equal to the hard threshold D, and for eachiteration, the higher hard threshold is increased according to

D _(high) =D _(high)+ε  (29)

For each iteration, the number of previous alerts potentially decreases.

FIG. 11 shows the time-series plot of data shown in FIG. 2 with the hardthreshold increased by the parameter ε to a higher hard thresholdrepresented by dashed line 1102. As a result, not all of the data valuesin the set of consecutive data 218 are greater than the higher hardthreshold 1102. As a result, the alert associated with the set 218 iseliminated and the feedback statistics associated with the alert areremoved from the set of feedback statistics F(B_(al)).

After each iterative increase of the hard threshold, the time-seriesdata is reexamined to reform the set of feedback statistics. Feedbackstatistics collected after previous alerts that were associated withdata above a previous hard threshold but are not above a current higherhard threshold are removed from the set of feedback statistics to give areduced set of feedback statistics associated with the higher hardthreshold

F _(J)(B _(al))≡{f ₁(B _(al)), . . . ,f _(J)(B _(al))}  (30)

where J is the number of all ratings (i.e., J≦K).

The set of feedback statistics F_(J)(B_(al)) includes only the feedbackstatistics associated with alerts that would have been generated by datagreater than the higher hard threshold.

Next, the weighted statistics for the set F_(J)(B_(i)) are recalculatedaccording to Equation (4) to give a set of weighted statistic

S _(J)(B _(i))≡={S(f ₁(B _(al))), . . . ,S(f _(J)(B _(al)))}  (31)

The weighted statistics histogram is generated from the set S_(J)(B_(i)) to give a set of normalized frequencies {h″_(r)} determinedfrom the l subintervals of the histogram. The entropy is calculated forthe normalized frequencies according to

$\begin{matrix}{{H\left( {{\overset{\_}{S}}_{j}\left( B_{al} \right)} \right)} = {- {\sum\limits_{r = 1}^{l}{h_{r}^{''}\log_{l}h_{r}^{''}}}}} & (32)\end{matrix}$

When H(S _(J)(B_(al)))≦U_(th), the confidence that includescontributions from feedback statistics associated with the new alerts iscalculated according to

C(B _(al))=1−H( S _(J)(B _(al)))  (33)

Otherwise, the confidence is given by

C(B _(al))=0  (34)

The average of the weighted statistics in the set S _(J)(B_(al)) iscalculated according to

$\begin{matrix}{{m_{al}\left( h_{\max} \right)} = {\frac{1}{J}\left( {\sum\limits_{j = 1}^{J}{S\left( {f_{j}\left( B_{al} \right)} \right)}} \right)}} & (35)\end{matrix}$

A noise degree (1−m_(al)(h_(max))) is calculated for the belief B_(al)and compared with the user's noise tolerance N. When the condition givenby Equation (18) is satisfied, the hard threshold is increased againaccording to Equation (29) and the operations described for Equations(30)-(35) are repeated for the higher hard threshold. On the other hand,when the condition given by Equation (16) is satisfied or after a fixednumber iterations P, the adjusted hard threshold is calculated accordingto

D=(1−C(B _(al)))D+C(B _(al))D _(high)  (36)

It should be noted that if in increasing the hard threshold, results inthe minimum number of alerts is less than a or the feedback statisticsfails to satisfy the condition given by Equation (14), the procedurestops.

The criticality L, timeliness w, and duration c are also adjusted basedon criticality confidence C(B_(crit)), timeliness confidenceC(B_(time)), and duration confidence C(B_(dut)). When the hard thresholdis left unchanged and satisfies the condition given in Equation (16),the confidences C(B_(crit)), C(B_(time)), and C(B_(dut)) are calculatedaccording to Equations (9). When the hard threshold is decreasedaccording to Equation (20), the confidences C(B_(crit)), C(B_(time)),and C(B_(dut)) are calculated according to Equations (9) and (10). Whenthe hard threshold is increased according to Equation (29), theconfidences C(B_(crit)), C(B_(time)), and C(B_(dut)) are calculatedaccording to Equations (9) and (10).

When the criticality confidence C(B_(crit))>0, the average of theweighted statistics m_(crit)(h_(max)) is calculated and the criticalityL is updated according to

L=(1−C(B _(crit)))L+C(B _(crit))·m _(crit)(h _(max))  (37)

Otherwise, the criticality is left unchanged. When L=0, the alert ischanged to “non-critical” or “none;” when 0<L≦0.25 the alert is changedto “informative;” when 0.25<L≦0.5 the alert is changed to “warning;”when 0.5<L≦0.75 the alert is changed to “immediate;” and when 0.75<L≦1the alert is changed to “critical.”

When the timeliness confidence C(B_(time))>0, the average of theweighted statistics m_(time)(h_(max)) is calculated. The wait time mayinitially be set to w=0.5. Assume that adjusting the wait time iscontrolled by a fraction k_(wait). In general, the fraction k_(wait) canvary within the interval [0,+∞), or, in particular, within the interval[0,1]. In other words, the wait time w varies from w−k_(wait)w tow+k_(wait)w under the condition that if w−k_(wait)w<0, then wait time isset to 0. The interval [w−k_(wait)w,w+k_(wait)w] is mapped to theinterval [0,1] by a linear function given by

$\begin{matrix}{y = {{f(x)} = {{\frac{1}{2k_{wait}w}x} - \frac{w - {k_{wait}w}}{2k_{wait}w}}}} & (38)\end{matrix}$

The wait time is updated according to Equation (38) by setting

y=(1−C(B _(time)))0.5+C(B _(time))·m _(time)(h _(max))  (39)

and taking the integer part to obtain w. Otherwise, when C(B_(time))=0,the wait time is left unchanged.

When the duration confidence C(B_(dur))>0, the average of the weightedstatistics m_(dur)(h_(max)) is calculated. The duration may initially beset to c=0.5. Assume that adjusting the duration is controlled by afraction k_(dur). In general, the fraction k_(dur) can vary within theinterval [0,+∞), or, in particular, within the interval [0,1]. In otherwords, the duration c varies from c−k_(dur)c to c+k_(dur)c under thecondition that if c−k_(dur)c<0, then the duration c is set to 0 Theinterval [c−k_(dur)c,c+k_(dur)c] is mapped to [0,1] by a linear functiongiven by

$\begin{matrix}{z = {{h(x)} = {{\frac{1}{2k_{dur}c}x} - \frac{c - {k_{dur}c}}{2k_{dur}c}}}} & (40)\end{matrix}$

The duration is updated according to Equation (40) by setting

z=(1−C(B _(dur)))0.5+C(B _(dur))·m _(dur)(h _(max))  (41)

and taking the integer part to obtain c. Otherwise, when durationconfidence C(B_(dur))=0, the duration is left unchanged.

FIG. 12 shows a flow-control diagram of a method for adjusting an upperhard threshold D. In block 1201, alert thresholds are initials. Forexample, a user may initially set the wait time w and duration c to 0.5and set the criticality L to a value in the interval [0,1]. The user mayalso set values for a noise tolerance N, an upper hard threshold D, atolerance parameter δ, and a step size ε. In block 1202, a time-seriesdata for a resource is continuously collected as described above withreference to FIG. 2. In decision block 1203, when the data is greaterthan the hard threshold as described above with reference to FIG. 2,control flows to block 1204. Otherwise, control flows to block 1202 anddata continues to be collected. In block 1204, an alert is generated. Inblock 1205, an alert count num-alerts is incremented. In block 1206,feedback statistics are collected from the user. The feedback statisticscan answers to survey questions as described above with reference toFIG. 3 or obtained by monitoring the user's action after the ispresented with an alert. In block 1207, a routine “calculate confidence”is called to calculate a confidence as described below with reference toFIG. 13. In decision block 1208, when number of alerts num_alerts isgreater than the minimum number of alerts a control flows to decisionblock 1209. Otherwise, control flows to decision block 1210. Decisionblock 1209 determines whether or not enough feedbacks statistics havebeen collected according to Equation (14). When enough feedbackstatistics have been collected according to Equation (14), control flowsto block 1211. Otherwise, control flows to decision block 1210. Indecision block 1201, as long as the data continues to be monitored, thecomputational operations in blocks 1202-1209 are repeated. In block1211, a routine “adjust alert thresholds” is called as described belowwith reference to FIG. 14.

FIG. 13 shows a flow-control diagram of the routine “calculateconfidences” called in block 1207 of FIG. 12. In block 1301, a set ofbeliefs B given by Equation (1) and set of feedback statisticsrepresented by Equation (2) are received. In block 1302, a for-looprepeats the computational operations of blocks 1303-1305 for eachbelief. In block 1303, a for-loop repeats the computational operationsof blocks 1304 and 1305 for each feedback statistic f_(k)(B_(i)) in aset of feedback statistics F(B_(i)) described above with reference toEquation (2). In block 1304, a weighted statistic S(f_(k)(B_(i))) iscalculated according to Equation (4). In block 1305, the method repeatsthe computational operation of block 1304 for another feedback statisticin the set F(B_(i)) until a weighted statistic has been calculated foreach of the feedback statistics in the set F(B_(i)). The weightedstatistic calculated according to blocks 1304 and 1305 form a set ofweighted statistics S(B_(i)) as described above with reference toEquation (7). In block 1306, normalized frequencies are calculated forthe set of weighted statistics based on a resolution l of the feedbackstatistics, as described above with reference to FIG. 6. In block 1307,the entropy H(S(B_(i))) of the set of weighted statistics is calculatedbased on the normalized frequencies according to Equation (8). Indecision block 1308, when the entropy H(S(B_(i))) is less than anuncertainty threshold U_(th), control flows to block 1309, otherwise,control flows to block 1310. The uncertainty threshold may be theuncertainty threshold given in Equation (11). In block 1309, aconfidence value C(B_(i)) may be calculated according to Equation (9)described above. In block 1310, the confidence value C(B_(i)) is set tozero. In decision block 1311, the method repeats the computationaloperations of blocks 1303-1311 for another belief until a confidence hasbeen calculated for each of the beliefs.

FIG. 14 shows a flow-control diagram of the routine “adjust alertthresholds” called in block 1211 of FIG. 12. When the confidenceC(B_(al)) is greater than zero in decision block 1401, control flows toblock 1402 in which the average of weighted statistics m_(al)(h_(max))is calculated. Otherwise, control flows to decision block 1407. Indecision block 1403, when the condition represented by Equation (17) issatisfied, control flows to block 1404. Otherwise, control flows todecision block 1405. In decision block 1405, when the conditionrepresented by Equation (18) is satisfied, control flows to block 1406.Otherwise, control flows to decision block 1407. When the results ofboth decision blocks 1403 and 1405 are “no,” the hard threshold is notadjusted, which is equivalent to satisfying the condition represented byEquation (16). In block 1405, a routine “move hard threshold down” iscalled as described below with reference to FIG. 15. In block 1406, aroutine “move hard threshold up” is called as described below withreference to FIG. 16. The routines called in blocks 1404 and 1406 bothcalculate confidences C(B_(crit)), C(B_(time)), and C(B_(dur))associated with either moving the hard threshold down or up as describedabove with reference to Equation (25) and Equation (34). In decisionblock 1407, when the criticality confidence C(B_(crit)) is greater thanzero, control flows block 1408 in which a routine “adjust criticality”is called as described below with reference to FIG. 17. Otherwise,control flows to decision block 1409 and the criticality is notadjusted. In decision block 1409, when timeliness confidence C(B_(time))is greater than zero, control flows to block 1410. Otherwise, controlflows to decision block 1412 and the timeliness w is not adjusted. Inblock 1410, the average of weighted statistics for timelinessm_(time)(h_(max)) is calculated. In block 1411, the timeliness isadjusted as described above with reference to Equation (40). In decisionblock 1412, when duration confidence C(B_(dur)) is greater than zero,control flows to block 1413. Otherwise, the duration c is not adjusted.In block 1413, the average of weighted statistics for durationm_(dur)(h_(max)) is calculated. In block 1414, the duration is adjustedas described above with reference to Equation (42).

FIG. 15 shows a flow-control diagram for the routine “move hardthreshold down” called in block 1404 of FIG. 14. In block 1501, feedbackstatistics associated with related alerts are collected as describedabove with reference to FIG. 7. In block 1502, an average feedback countis calculated according to Equation (19). In block 1503, the hardthreshold is decreased as described above with reference to Equation(20). In decision block 1504, when additional alerts are identified asdescribed above with reference to FIG. 8, control flows to block 1505.Otherwise, control flows back to block 1503. In block 1505, a lowerbound V(B_(i)) for the modal subinterval of the weighted statisticshistogram is identified. In block 1506, estimated feedback statisticsfor new alerts are added to the set of feedback statistics as describedabove with reference to FIG. 9 and Equation (21). In block 1507, theaverage of weighed statistics m_(al)(h_(max)) is calculated according toEquation (23). In decision block 1508, when the condition represented byEquation (17) is satisfied, control flows to decision block 1509.Otherwise, control flows to block 1510. In decision block 1509, when thenoise degree is a maximum is close to the noise tolerance N, controlflow to block 1510. Otherwise control flows to decision block 1511. Inblock 1510, an adjusted hard threshold is calculated according toEquation (24).

The control-flow diagram in FIG. 15 also includes blocks 1511-1515 thatmay be used to further decrease the hard threshold. In block 1511, whenthe condition given by Equation (25) is satisfied, control flows toblock 1512. Otherwise, control flows to block 1510. The lesser intervalnext to the modal interval is identified in block 1512 and the lowerbound of the lesser interval is identified in block 1513, as describedabove with reference to FIG. 10. In block 1514, the confidence C(B_(al))is calculated according to Equation (27). In decision block 1515, whenthe confidence C(B_(al)) equals zero, control flows to block 1503.Otherwise, control flows to block 1506.

FIG. 16 shows a flow-control diagram for the routine “move hardthreshold up” called in block 1406 of FIG. 14. In block 1601, the hardthreshold is increased as described above with reference to Equation(30). In block 1602, the set of feedback statistics is reduced byremoving feedback statistics associated with deleted alerts as describedabove with reference to Equation (31). In block 1603, the confidenceC(B_(al)) is calculated according to Equation (33). In decision block1604, when the alert confidence C(B_(al)) is greater than zero, controlflows to block 1605 in which the average of weighted statistics for theindicativeness of the alerts is calculated. Otherwise, control flowsback to block 1601 and the hard threshold is increased. In block 1606,when the condition represented by Equation (18) is satisfied, controlflows to block 1608. Otherwise, control flows to block 1607 in which anadjusted hard threshold is calculated according to Equation (36). Indecision block 1608, when the noise degree is a maximum is close thenoise tolerance N, control flow to block 1607. Otherwise control flowsto decision block 1601.

FIG. 17 shows a flow-control diagram for the routine “adjustcriticality” called in block 1408 of FIG. 14. In bock 1701, average ofthe weighted statistics for criticality confidence C(B_(crit)) iscalculated. In block 1702, the criticality is calculated according toEquation (38). In decision block 1703, when L=0, the alert is changed to“non-critical” or “none” in block 1704. In decision block 1705, when0<L≦0.25 the alert is changed to “informative” in block 1706. Indecision block 1707, when 0.25<L≦0.5 the alert is changed to “warning”in block 1708. In decision block 1709, when 0.5<L≦0.75 the alert ischanged to “immediate” in block 1710. Otherwise, and the alert ischanged to “critical” in block 1711.

Although FIGS. 12-17 present flow-control diagrams of a method foradjusting an upper hard threshold D, methods and systems are notintended to be limited to adjusting upper hard thresholds. The methodsdescribed above may also be used to adjust a lower hard threshold d,such example lower hard threshold d in FIG. 2. The noise degreecalculated as described above with reference to Equation (15) andcompared with the user's selected noise tolerance N_(lower), which candiffer from the noise tolerance N for the upper threshold, to determinewhether or not the alerts generated by violating the lower hardthreshold d satisfy the user's tolerance for false positive alerts. Whenthe difference between the noise degree and the user's noise toleranceN_(lower) satisfies the condition given by Equation (16) with N equal toN_(lower), the hard threshold d is left unchanged. However, when thedifference satisfies the condition represented by Equation (18), with Nequal to N_(lower), the noise degree is too high, and the hard thresholdis decreased according to

d _(high) =d _(high)−ε  (43)

On the other hand, when the difference satisfies the conditionrepresented by Equation (17), with N equal to N_(lower), the noisedegree is too low and the threshold is increased according to

d _(low) =d _(low)+ε  (44)

Although the control-flow diagrams in FIGS. 14-16 are directed toadjusting alert thresholds for an upper threshold D, these control-flowdiagrams can be modified for adjusting alert thresholds for a lower hardthreshold d. The flow-control diagram in FIG. 14 is replaced byflow-control diagram in FIG. 18; the flow-control diagram in FIG. 15 isreplaced by flow-control diagram in FIG. 19; and the flow-controldiagram in FIG. 16 is replaced by flow-control diagram in FIG. 20. Notethat in decision blocks 1801 and 1803 of FIG. 18, decision blocks 1902and 1903 of FIG. 19, and decision blocks 2002 and 2004 of FIG. 20, thenoise tolerance N is replaced by the noise tolerance N_(lower). In block1802 of FIG. 18, a routine “move threshold up” is called and implementedas represented in FIG. 19, and in block 1804 of FIG. 18, a routine “movethreshold down” is called an implemented as represented in FIG. 20. Inblock 1901 of FIG. 19, the lower threshold is adjusted according toEquation (44) and an adjusted threshold is calculated in block 1904according to

d=(1−C(B _(al)))d+C(B _(al))d _(low)  (45)

In block 2001 of FIG. 20, the lower threshold is adjusted according toEquation (43) and an adjusted threshold is calculated in block 2003according to

d=(1−C(B _(al)))d+C(B _(al))d _(high)  (46)

It is appreciated that the various implementations described herein areintended to enable any person skilled in the art to make or use thepresent disclosure. Various modifications to these implementations willbe readily apparent to those skilled in the art, and the genericprinciples defined herein may be applied to other implementationswithout departing from the spirit or scope of the disclosure. Forexample, any of a variety of different implementations can be obtainedby varying any of many different design and development parameters,including programming language, underlying operating system, modularorganization, control structures, data structures, and other such designand development parameters. Thus, the present disclosure is not intendedto be limited to the implementations described herein but is to beaccorded the widest scope consistent with the principles and novelfeatures disclosed herein.

1. A method stored in one or more data-storage devices and executedusing one or more processors of a computing environment to adjust hardthresholds based on user feedback, the method comprising: generatingalerts when time-series data generated by a data-generating entityviolates a hard threshold; collecting user feedback from a surveypresented to a user of the data-generating entity following each of thealerts; and when a number of user feedbacks generated by the user isgreater than an average number of feedbacks per user of thedata-generating entity, adjusting the hard threshold based on the userfeedback.
 2. The method of claim 1, wherein generating alerts when thetime-series data violates the hard threshold further comprises one ofwhen the hard threshold is an upper hard threshold, generating an alertwhen a portion of the data is greater than the upper hard threshold; andwhen the hard threshold is a lower hard threshold, generating an alertwhen a portion of the data is less than the lower hard threshold.
 3. Themethod of claim 1, wherein collecting user feedback comprises presentingthe user with survey questions regarding indicativeness, criticality,timeliness, and duration of each alert.
 4. The method of claim 1,wherein adjusting the hard threshold based on the user feedbackcomprises: calculating an alert confidence value based on the userfeedback; calculating an adjusted hard threshold from the hard thresholdand a step size value greater than zero when the alert confidence isgreater than zero; and setting the hard threshold equal to the adjustedhard threshold.
 5. The method of claim 4, wherein calculating the alertconfidence value comprises: determining feedback statistics from valuesassigned to user feedback regarding indicativeness, criticality,timeliness, and duration of each alert; calculating weighted statisticsfrom the feedback statistics; forming a histogram of the weightedstatistics distributed over a number of subintervals; calculatingnormalized frequencies of the weighted statistics based on thedistribution of the weighted statistics; calculating an entropy value ofthe weighted statistics; and calculating a confidence value based on theentropy value of the weighted statistics.
 6. The method of claim 5wherein determining feedback statistics comprises generating sets ofuser feedback statistics regarding criticality, timeliness, and durationof the number of alerts based on the user feedback; calculating acriticality confidence, timeliness confidence, and duration confidencebased on corresponding feedback statistics; calculating adjustedcriticality when the criticality confidence is greater than zerocalculating adjusted timeliness when the timeliness confidence isgreater than zero; and calculating adjusted duration when the durationconfidence is greater than zero.
 7. The method of claim 4, whereincalculating the adjusted hard threshold further comprises calculating anaverage of weighted statistics based on the feedback statistics when thealert confidence is greater than zero; calculating a noise degree fromthe average of the weighted statistics; when the hard threshold is anupper hard threshold, decreasing the hard threshold, when a differencebetween the noise degree and a user-defined noise tolerance is negativevalued and outside a tolerance interval; increasing the hard threshold,when the difference between the noise degree and the user-defined noisetolerance is positive valued and outside the tolerance interval; andcalculating the adjusted hard threshold as a function of the average ofthe weighted statistics, the alert confidence, and one of the increasedand decreased hard threshold.
 8. The method of claim 4, whereincalculating the adjusted hard threshold further comprises: calculatingan average of weighted statistics based on the feedback statistics whenthe alert confidence is greater than zero; calculating a noise degreefrom the average of the weighted statistics; when the hard threshold isa lower hard threshold, increasing the hard threshold, when a differencebetween the noise degree and a user-defined noise tolerance is negativevalued and outside a tolerance interval; decreasing the hard threshold,when the difference between the noise degree and the user-defined noisetolerance is positive valued and outside the tolerance interval; andcalculating the adjusted hard threshold as a function of the average ofthe weighted statistics, the alert confidence, and one of the increasedand decreased hard threshold.
 9. A system to adjust a hard threshold ofa data-generating entity comprising: one or more processors; one or moredata-storage devices; and a routine stored in the data-storage devicesthat when executed using the one or more processors performs generatingalerts when time-series data generated by a data-generating entityviolates a hard threshold; collecting user feedback from a surveypresented to a user of the data-generating entity following each of thealerts; and when a number of user feedbacks generated by the user isgreater than an average number of feedbacks per user of thedata-generating entity, adjusting the hard threshold based on the userfeedback.
 10. The system of claim 9, wherein generating alerts when thetime-series data violates the hard threshold further comprises one ofwhen the hard threshold is an upper hard threshold, generating an alertwhen a portion of the data is greater than the upper hard threshold; andwhen the hard threshold is a lower hard threshold, generating an alertwhen a portion of the data is less than the lower hard threshold. 11.The system of claim 9, wherein collecting user feedback comprisespresenting the user with questions regarding indicativeness,criticality, timeliness, and duration of each alert.
 12. The system ofclaim 9, wherein adjusting the hard threshold based on the user feedbackcomprises: calculating an alert confidence value based on the userfeedback; calculating an adjusted hard threshold from the hard thresholdand a step size value greater than zero when the alert confidence isgreater than zero; and setting the hard threshold equal to the adjustedhard threshold.
 13. The system of claim 12, wherein calculating thealert confidence value comprises: determining feedback statistics fromvalues assigned to user feedback regarding indicativeness, criticality,timeliness, and duration of each alert; calculating weighted statisticsfrom the feedback statistics; forming a histogram of the weightedstatistics distributed over a number of subintervals; calculatingnormalized frequencies of the weighted statistics based on thedistribution of the weighted statistics; calculating an entropy value ofthe weighted statistics; and calculating a confidence value based on theentropy value of the weighted statistics.
 14. The system of claim 13wherein determining feedback statistics comprises generating sets ofuser feedback statistics regarding criticality, timeliness, and durationof the number of alerts based on the user feedback; calculating acriticality confidence, timeliness confidence, and duration confidencebased on corresponding feedback statistics; calculating adjustedcriticality when the criticality confidence is greater than zerocalculating adjusted timeliness when the timeliness confidence isgreater than zero; and calculating adjusted duration when the durationconfidence is greater than zero.
 15. The system of claim 12, whereincalculating the adjusted hard threshold further comprises calculating anaverage of weighted statistics based on the feedback statistics when thealert confidence is greater than zero; calculating a noise degree fromthe average of the weighted statistics; when the hard threshold is anupper hard threshold, decreasing the hard threshold, when a differencebetween the noise degree and a user-defined noise tolerance is negativevalued and outside a tolerance interval; increasing the hard threshold,when the difference between the noise degree and the user-defined noisetolerance is positive valued and outside the tolerance interval; andcalculating the adjusted hard threshold as a function of the average ofthe weighted statistics, the alert confidence, and one of the increasedand decreased hard threshold.
 16. The system of claim 12, whereincalculating the adjusted hard threshold further comprises: calculatingan average of weighted statistics based on the feedback statistics whenthe alert confidence is greater than zero; calculating a noise degreefrom the average of the weighted statistics; when the hard threshold isa lower hard threshold, increasing the hard threshold, when a differencebetween the noise degree and a user-defined noise tolerance is negativevalued and outside a tolerance interval; decreasing the hard threshold,when the difference between the noise degree and the user-defined noisetolerance is positive valued and outside the tolerance interval; andcalculating the adjusted hard threshold as a function of the average ofthe weighted statistics, the alert confidence, and one of the increasedand decreased hard threshold.
 17. A non-transitory computer-readablemedium encoded with machine-readable instructions that implement amethod carried out by one or more processors of a computer system toperform the operations of generating alerts when time-series datagenerated by a data-generating entity violates a hard threshold;collecting user feedback from a survey presented to a user of thedata-generating entity following each of the alerts; and when a numberof user feedbacks generated by the user is greater than an averagenumber of feedbacks per user of the data-generating entity, adjustingthe hard threshold based on the user feedback.
 18. The medium of claim17, wherein generating alerts when the time-series data violates thehard threshold further comprises one of when the hard threshold is anupper hard threshold, generating an alert when a portion of the data isgreater than the upper hard threshold; and when the hard threshold is alower hard threshold, generating an alert when a portion of the data isless than the lower hard threshold.
 19. The medium of claim 17, whereincollecting user feedback comprises presenting the user with questionsregarding indicativeness, criticality, timeliness, and duration of eachalert.
 20. The medium of claim 17, wherein adjusting the hard thresholdbased on the user feedback comprises: calculating an alert confidencevalue based on the user feedback; calculating an adjusted hard thresholdfrom the hard threshold and a step size value greater than zero when thealert confidence is greater than zero; and setting the hard thresholdequal to the adjusted hard threshold.
 21. The medium of claim 20,wherein calculating the alert confidence value comprises: determiningfeedback statistics from values assigned to user feedback regardingindicativeness, criticality, timeliness, and duration of each alert;calculating weighted statistics from the feedback statistics; forming ahistogram of the weighted statistics distributed over a number ofsubintervals; calculating normalized frequencies of the weightedstatistics based on the distribution of the weighted statistics;calculating an entropy value of the weighted statistics; and calculatinga confidence value based on the entropy value of the weightedstatistics.
 22. The medium of claim 21, wherein determining feedbackstatistics comprises generating sets of user feedback statisticsregarding criticality, timeliness, and duration of the number of alertsbased on the user feedback; calculating a criticality confidence,timeliness confidence, and duration confidence based on correspondingfeedback statistics; calculating adjusted criticality when thecriticality confidence is greater than zero calculating adjustedtimeliness when the timeliness confidence is greater than zero; andcalculating adjusted duration when the duration confidence is greaterthan zero.
 23. The medium of claim 20, wherein calculating the adjustedhard threshold further comprises calculating an average of weightedstatistics based on the feedback statistics when the alert confidence isgreater than zero; calculating a noise degree from the average of theweighted statistics; when the hard threshold is an upper hard threshold,decreasing the hard threshold, when a difference between the noisedegree and a user-defined noise tolerance is negative valued and outsidea tolerance interval; increasing the hard threshold, when the differencebetween the noise degree and the user-defined noise tolerance ispositive valued and outside the tolerance interval; and calculating theadjusted hard threshold as a function of the average of the weightedstatistics, the alert confidence, and one of the increased and decreasedhard threshold.
 24. The medium of claim 20, wherein calculating theadjusted hard threshold further comprises: calculating an average ofweighted statistics based on the feedback statistics when the alertconfidence is greater than zero; calculating a noise degree from theaverage of the weighted statistics; when the hard threshold is a lowerhard threshold, increasing the hard threshold, when a difference betweenthe noise degree and a user-defined noise tolerance is negative valuedand outside a tolerance interval; decreasing the hard threshold, whenthe difference between the noise degree and the user-defined noisetolerance is positive valued and outside the tolerance interval; andcalculating the adjusted hard threshold as a function of the average ofthe weighted statistics, the alert confidence, and one of the increasedand decreased hard threshold.